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1 .  SUMMARY 


1.1  Report  Period 

The  seventh  quarter  of  contract  activity  involved  (1)  implementation 
of  technical  changes  on  the  system,  (2)  development  of  plans  for  experimental 
evaluation  of  the  system,  and  (3)  demonstration  of  the  system  to  planning 
groups  at  NADC.  The  following  tasks  were  completed  during  the  report 
period: 

(1)  A  series  of  important  changes  were  made  to  improve 
system  flexibility  and  speed,  and  to  ready  the  system 
for  experimental  studies.  Included  in  the  modifications 
were:  (a)  algorithms  for  attribute  definition  and 
weighting,  (b)  tree  traversing  programs,  and  (c)  a  new 
set  of  confliction  identification  and  resolution 
procedures. 

(2)  Plans  for  a  series  of  experimental  evaluations  have 
been  finalized.  The  evaluations  will  center  on 
determining  the  effectiveness  of  different  procedures 
for  conflict  resolution  and  problem  definition,  and 
will  examine  the  relative  performance  of  aided  and 
non-aided  groups. 

(3)  A  full  scale  system  demonstration  was  performed  at 
NADC  in  Warminister,  Pennsylvania.  Several  high 
level  decision  making  groups  were  given  an 
opportunity  to  use  the  system.  Virtually  all  of 
the  personnel  were  impressed  with  the  aiding 
capabilities  provided. 
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1.2 


Next  Period 


The  contract  period  during  the  next  quarter  will  primarily 
concentrate  on  system  evaluation  studies.  The  specific  items  of  work 
for  the  next  period  include: 

(1)  Perform  full  scale  experimental  studies  of  system 
aiding. 

(2)  Design  system  integration  with  CACI  Executive  Aid. 

(3)  Continue  cooperative  effort  with  NADC  to  develop 
system  design  application. 

1.3  Program  Milestones 

The  milestone  chart  for  the  contract  program  is  shown  in  Figure 
1-1,  with  the  report  period  illustrated  as  the  shaded  portion. 
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Correction  Procedures 


TABLE  1-1.  PROGRAM  MILESTONES 


2.  PROGRAM  OVERVIEW 


2.1  Statement  of  Problem 


Constant  escalation  in  weapons  cost  and  effectiveness,  as  well  as 
increasing  complexity  of  international  relations,  makes  military  decision 
making  more  critical  today  than  ever  before.  In  today's  military 
environment,  most  upper-level  decisions  are  made  by  committees  and  staff 
groups.  Typically,  such  groups  contain  experts  from  several  speciality 
areas,  who  bring  to  the  decision  environment  disparate  sets  of  values. 
Decision  time  is  usually  limited,  the  decision  making  procedure  is 
relatively  unstructured,  and  intragroup  conflicts  arise  on  a  broad 
variety  of  issues.  Consequently  the  group  is  unable  to  consider  the 
maximum  set  of  alternatives,  conflicts  are  not  resolved  in  an  optimum 
manner,  and  the  resultant  decision  is  rarely  up  to  the  aggregate  potential 
of  the  group  membership. 

2.2  Rationale 


Decision  analysis  offers  a  promising  approach  to  solving  these 
problems.  The  analytical  procedure  of  building  a  decision  tree  formalizes 
the  decision  process,  and  permits  incorporation  of  individual  values 
(utilities)  into  the  selection  of  alternative  courses  of  action  (Hays, 
O'Connor,  and  Peterson,  1975).  However,  decision  analysis  as  it  is 
usually  practiced,  is  a  highly  personal  and  time-consuning  process. 
Decision  analysts  are  often  called  upon  to  assist  in  the  solution  of 
problems  ranging  over  a  large  variety  of  domains.  In  most  cases,-  the 
decision  analysts  know  far  less  about  the  problem  domain  than  do  their 
clients.  Thus,  their  contributions  are  confined  primarily  to  the  phases 
of  formalization  and  optimization.  While  optimization  is  usually  computer 
assisted,  the  formalization  phase  invariably  has  been  accomplished 
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manually,  using  lengthy  interviews  of  persons  more  familiar  with  the 
problem  area.  This  approach  is  generally  incompatible  with  the  conditions 
of  command  group  decision  making. 

Accordingly,  it  would  be  highly  worthwhile  to  automate  the 
formalization  phase,  using  an  interactive  computer  system  to  interrogate 
the  group  members  and  to  construct  a  decision  tree  based  on  their  responses. 
The  purpose  of  the  research  undertaken  here  is  to  develop  and  evaluate 
the  means  by  which  such  an  interactive  aid  could  be  used  to  improve  group 
decision  making. 

2.3  Objectives 


The  goal  of  the  research  program  addressed  in  this  progress  report 
is  to  develop  an  automated  decision  tree  elicitation  system  using  on¬ 
line  sensitivity  analysis  with  direct  real-time  group  feedback  and 
evaluate  its  effectiveness  in  aiding  group  decision  making. 

The  specific  objectives  of  the  current  program  include  the 
following: 

(1)  Develop  computer  programs  for  efficient,  comprehensive, 
elicitation  of  decision  trees  from  a  decision  making 
group. 

(2)  Develop  computer  programs  for  identifying  structural  and 
numerical  differences  among  the  contributions  of 
individual  group  members,  for  merging  these  contributions 
and  for  resolving  the  points  of  conflict. 

(3)  Develop  effective  means  for  displaying  to  the  group 
the  results  of  the  elicitation  procedures  and  conflict 
analyses. 
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(4)  Integrate  the  various  programs  and  techniques  into  a 
complete  aiding  system  which  can  be  readily  transferred 
to  other  test  environments. 

(5)  Experimentally  test  the  Group  Decision  Aid,  using  a 
variety  of  representative  military  decision  problems, 
to  demonstrate  its  advantages  under  realistic 
conditions  of  use. 

(6)  On  the  basis  of  the  developmental  effort  and  the 
experimental  results,  establish  guidelines  and 
recommendations  for  future  military  applications 
of  the  group  decision  aiding  methodology. 
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3.  SYSTEM  IMPROVEMENTS 


3.1  Overview 


The  last  quarter  of  activity  involved  completion  of  a  number  of 
system  refinements  and  improvements.  These  improvements  were  necessary 
to  ready  the  system  for  major  experimental  studies.  Among  the  modifications 
made  were  changes  to  attribute  definition  and  weighting  procedures,  the 
tree  traversing  program,  the  conflict  identification  and  resolution 
algorithms,  and  the  feedback  options.  The  changes  have  led  to  greater 
generality,  flexibility  and  speed  of  the  system.  Each  of  the  improvements 
will  be  detailed  in  the  following  sections. 

3.2  Attribute  Definition  and  Weighting 

A  series  of  changes  were  made  recently  to  facilitate  the  definition, 
scaling,  weighting  and  revision  of  the  attribute  set.  The  specific 
procedures  -  selecting  a  "baseline"  attribute,  comparison  of  other  attributes 
to  this  attribute,  etc.  -  are  detailed  in  the  previous  progress  report 
(Leal,  et  al  1978).  Preliminary  tests  of  the  procedures  with  groups  at 
Perceptronics  and  at  NADC  (see  section  5)  resulted  in  the  refinements 
listed  below. 

Two  frames  were  found  to  be  necessary  for  display  of  conflicts 
regarding  attribute  weights.  A  graphical  display  of  normalized  weights 
(in  which  Ew^  =  100  for  each  participant)  was  found  to  b'  necessary  to 
identify  and  rank  conflicts  (Figure  3-1).  Following  this  frame  s  display 
of  the  raw  weights  (Figure  3-2)  was  found  to  best  for  re-entry  of  weight 
estimates,  since  the  weights  are  again  estimated  in  comparison  to  the 
baseline  weight. 
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A1  COST  A3  IMAGE 

A2  RISK _ A4  RETURN 


FIGURE  3-1.  RAW  WEIGHT  CONFLICT  FRAME 


A1  COST  A3  IMAGE 

A2  RISK  A4  RETURN 


FIGURE  3-2.  NORMALIZED  WEIGHT  CONFLICT  FRAME 
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Some  difficulties  were  noted  at  NADC  regarding  the  procedures 
for  revision  of  the  attribute  set.  Frequent  revision  of  the  set  of 
attributes  was  found  to  be  time  consuming  and  difficult.  The  problem 
appeared  to  be  with  the  basic  philosophy  of  definition  of  the  attribute 
set,  however,  rather  than  with  the  procedures  for  revision.  For  example, 
several  of  the  NADC  groups  defined  different  atti .cute  sets  for  different 
portions  of  the  tree.  To  be  effective  as  a  tool  for  evaluation,  the 
attribute  set  should  derive  directly  from  the  set  of  objectives  for  the 
problem.  Each  attribute  should  reflect  the  degree  to  which  an  alternative 
or  outcome  achieves  a  specific  objective.  Comparability  across  choices 
demands  that  the  attribute  set  be  invariant,  or  that  changes  in  the  set 
be  incorporated  in  earlier  evaluations.  Subsequent  tests  of  the  system 
will  include  detailed  instructions  for  the  development  and  revision  of 
the  attribute  set. 

3.3  Tree  Traversing 

A  tree  traversing  program  was  implemented  to  allow  greater 
freedom  in  tree  development.  In  particular  this  program  allows  override 
of  the  sensitivity  analysis  program  regarding  next  node  selection.  The 
intermediator  calls  this  program  and  moves  through  the  tree  to  the  desired 
node  by  pressing  one  of  four  directional  buttons.  The  node  may  then  be 
opened  for  probability  and  value  estimation.  The  tree  traverser  may  also 
be  used  to  locate  a  node  and  delete  the  subtree  extending  from  that  node. 
This  function  is  useful  for  redefining  sections  of  the  tree.  An  additional 
capability  of  this  program  is  the  display  of  the  subtree  from  any  selected 
node. 

3.4  Conflict  Resolution  Algorithm 

The  original  conflict  identification  and  resolution  algorithm 
was  based  on  the  degree  of  difference  of  opinion  on  each  value  estimate. 
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If  the  differences  among  the  group  were  greater  than  a  threshold  amount, 
the  MAU  decomposition  procedure  was  invoked.  This  procedure  has  been 
found  to  be  effective  but  is  somewhat  arbitrarily  based.  For  example, 
a  large  difference  in  estimated  values  for  a  given  outcome  may  be  present 
without  a  conflict  regarding  the  preferred  course  of  action.  Conversely, 
a  sub-threshold  difference  in  values  may  actually  produce  a  conflict  at 
the  root  node. 


A  new  conflict  resolution  program  has  been  implemented.  This 
algorithm  identifies  conflicts  as  those  differences  in  values  which  cause 
a  conflict  in  the  initial  decision.  The  sensitivity  analysis  program 
(also  used  for  selecting  the  next  node  for  expansion)  tests  each 
participant’s  values,  one  at  a  time.  If  a  conflict  is  present,  the  node 
just  opened  is  decomposed  into  attributes,  and  estimates  of  each  attribute 
taken.  After  recombination,  the  overall  values  are  again  tested  for 
conflicts  at  the  root  decision.  If  a  conflict  is  still  present,  the 
multi-attribute  procedure  may  again  be  called. 

Using  this  procedure,  conflicts  can  be  identified  at  any  point 
after  the  first  level  of  the  tree  just  as  for  the  variability-based 
algorithm.  In  the  first  level  itself,  no  provisional  values  are  elicited. 
At  the  second  and  succeeding  levels,  each  set  of  overall  value  judgements 
for  a  given  node  is  rolled  back  to  determine  if  a  conflict  is  produced 
at  the  root  node.  If  a  conflict  is  present,  the  attribute  level  estimation 
frame  cycles  through  each  of  the  attributes  one  at  a  time.  Once  all  of 
the  attributes  are  estimated,  the  program  determines  the  aggregated  value 
for  each  participant  by  taking  the  weighted  sum  of  attributes: 


valuejn  = 


attributes 

i 
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Where 


value 


jk 


is  the  overall  value  of  action  k  to 


participant  j  W^-  the  importance  weight  of  attribute  i 
Aijk  is  the  level  of  attribute  i  associated  with 
action  k  by  participant  j. 


The  conflict  feedback  display  (Figure  3-3)  is  composed  of  the 
aggregated  values  for  the  participants  still  involved  in  a  root-decision 
conflict.  Instead  of  highlighting  those  attributes  having  high  variance, 
the  participants  are  shown  the  differences  in  level  and  the  importance 
weights  associated  with  each  attribute.  In  this  way,  the  participants 
can  get  an  idea  of  how  much  each  attribute  actually  contributes  to  the 
conflict  in  values.  A  subset  of  the  attributes  or  the  entire  set  may 
then  be  chosen  for  re-estimation.  If  a  conflict  is  still  present,  the 
process  may  be  repeated. 


Several  additional  procedural  options  have  been  implemented  for 
system  flexibility.  The  group  can  invoke  a  multi-attribute  decomposition 
even  if  there  is  no  conflict  in  the  overall  value  estimates.  This  is 
useful  if  the  participants  are  uncertain  as  to  the  value  of  a  complex 
outcome.  Also,  the  value  and  probability  estimates  can  be  re-entered 
prior  to  the  conflict  testing  should  there  be  a  mistake  in  input.  A 
special  display  is  provided  in  the  case  of  the  probability  estimates. 

The  group  is  displayed  the  mean  and  range  of  probability  estimates  and 
is  allowed  to  re-enter  if  necessary. 

The  audit  trail  has  been  expanded  to  include  information  about 
the  new  conflict  identification  processes.  A  ranking  and  evaluation  of 
alternatives  by  each  participant  will  be  taken  at  each  decision  node. 
Choices  made  at  each  point,  such  as  display  frames,  actions,  etc.,  will 
also  be  noted. 
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4.  EXPERIMENTAL  PLAN 


4.1  Overview 


A  series  of  experimental  evaluations  of  various  aspects  of  the 
group  decision  aid  have  been  planned.  The  experimental  study  will  be 
directed  toward  the  following  major  objectives. 

(1)  To  determine  the  relative  effectiveness  of  alternative 
procedures  for  problem  development  and  conflict 
resolution. 

(2)  To  determine  the  specific  contributions  to  problem 
definition,  conflict  resolution  and  decision  quality 
provided  by  the  group  decision  aiding  system. 

(3)  To  aid  in  development  of  guidelines  and  materials  for 
system  operation. 

These  objectives  are  planned  to  be  accomplished  through  an 
experimental  program  based  on  performance  comparisons  between  alternative 
system  configurations  and  between  aided  and  unaided  group  interaction. 

4.2  Experimental  Hypotheses 

The  experimental  hypotheses  deal  with  the  aiding  possible  under 
various  procedural  options.  The  hypotheses  are: 

(1)  Use  of  a  discussion  and  problem  definition  period  prior 
to  system  use  will  result  in  less  attribute  redefinition 
and  node  deletion,  greater  overall  tree  complexity,  and 
more  confidence  in  the  final  recommendation  than  immediate 
use  of  aiding  system. 
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(2)  The  root  decision  conflict  rule  will  result  in  a  lower 
number  of  identified  conflicts,  greater  tree  development, 
and  greater  acceptance  of  the  final  choice  than  the 
variability-based  rule. 

(3)  The  variability-based  conflict  identification  algorithm 
will  result  in  greater  agreement  and  confidence  in  the 
final  ranking  of  the  decision  options.  The  variability- 
based  rule  will  also  require  less  understanding  of  the 
aiding  procedure  than  the  root  decision  conflict  rule. 

(4)  Use  of  the  group  decision  aiding  system  with  either 
conflict  resolution  algorithm  will  result  in  greater 
speed,  tree  complexity,  confidence  in  the  final 
decision,  and  consideration  of  critical  decision  issues 
(as  identified  in  the  CACI  analysis  of  the  crisis 
scenario)  than  found  with  unstructured  group  performance. 

4.3  Experimental  Procedures 

The  scenario  to  be  used  for  the  evaluation  will  be  the  crisis 
scenario  based  on  counter  terrorist  actions.  This  CACI-developed  scenario 
has  plausibility,  reasonable  complexity,  numerous  judgemental  issues, 
and  has  been  the  subject  of  extensive  analysis.  Comprehensive  measures 
of  group  participation,  decision  development,  conflict  resolution, 
and  decision  quality  have  been  developed  with  respect  to  this  scenario. 

A  more  detailed  analysis  of  the  scenario  and  the  analysis  measures  are 
found  in  the  previous  progress  report  (Steeb,  et  al  1978). 

Experimental  Design 

The  experimental  variables  to  be  investigated  include  the  following 
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1)  Form  of  conflict  identification:  Two  levels,  variability- 
based  algorithm  and  root  decision  algorithm. 

2)  Group  preparation:  Two  levels,  use  of  group  aid  throughout 
problem  and  use  of  aiding  system  after  preliminary, 
unstructured  problem  definition. 

3)  Presence  of  aiding:  Two  levels,  use  of  group  decision  aiding 
system  vs.  unstructured  group  interaction. 

The  above  conditions  are  sufficient  for  testing  the  experimental 
hypotheses  and  represent  the  primary  system  application  issues  at  this 
point  of  development.  The  factors  will  be  incorporated  into  a  sequential 
experimental  design,  with  one  variable  to  be  examined  at  each  stage.  The 
conditions  included  at  each  subsequent  stage  will  be  determined  by  the 
previous  results.  The  variable  of  lesser  importance,  degree  of  group 
preparation,  will  be  examined  first.  Following  will  be  studies  of  aiding 
form  and  presence  of  the  aiding  system  itself. 


5.  SYSTEM  DEMONSTRATION 


A  full  scale  system  demonstration  was  performed  during  the  week 
of  September  18  at  NADC  in  Warminster,  Pennsylvania.  The  intent  of  the 
demonstration  was  to  familiarize  Navy  decision  makers  with  the  capabilities 
and  operation  of  the  system,  and  to  provide  a  forum  for  further  inputs 
regarding  Navy  needs. 

The  system  was  transferred  to  the  PDP  11/70  at  Warminster,  and 
connected  to  their  Advent  projection  system.  This  was  accomplished 
shortly  after  completion  of  all  of  the  changes  described  in  Chapter  3 
save  for  the  new  conflict  resolution  program. 

Three  groups  worked  with  the  system  at  NADC  representing  the  Naval 
Air  Development  Center  Management  Center  computer  and  software  directorate, 
and  project  design  group.  Questionnaires  administered  following  the 
demonstrations  revealed  the  following  reactions. 

1)  All  respondents  said  that  the  system  would  help  them  make 
better  decisions. 

2)  Most  respondents  felt  that  the  system  would  not  result  in 
faster  decisions,  mentioning  that  the  procedures  for  entry 
of  probabilities  and  utilities  were  too  time  consuming. 

3)  Most  felt  that  the  procedures  were  easy  to  understand, 
although  some  practice  time  was  necessary. 

4)  All  participants  were  impressed  with  the  nature  and 
effectiveness  of  the  group  interaction. 
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In  all,  the  participants  saw  many  advantages  in  the  system: 
explicit  decision  data,  bookkeeping  by  the  computer,  systematic 
analysis,  and  aids  to  visualization  and  interaction.  At  the  same  time, 
needs  for  streamlining  procedures  and  allowing  revision  of  the  inputs 
were  identified.  Several  procedural  changes,  notable  in  the  input 
processes,  were  made  in  the  system  in  response  to  these  suggestions. 
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